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Abstract 

In the last decades, the number of bilingual schools in Europe has increased sig¬ 
nificantly. Most of these schools implement CLIL methodology in their teaching 
and, consequently, there is a demand for new and appropriate materials for each 
content subject taught through a foreign language: new textbooks written in L2 
are needed, and the existent offer does not meet the current demand. Given that 
CLIL materials development is a relatively recent field, there is not much research 
on the evaluation of those still insufficient course books. In this paper, and in or¬ 
der to evaluate CLIL textbooks, we propose a tentative checklist twofold based: on 
the one hand, based on previous checklists created to evaluate ELT textbooks - CLIL 
materials use principles/techniques of foreign language teaching (Steirt and Mas- 
ser, 2011)-, and, on the other hand, based on the criteria for producing quality CLIL 
learning materials drawn up by Mehisto (2012). Further research is in process in 
order to refine the checklist with the results of an analysis of the criteria carried 
out by a focus group of CLIL teachers, and of a survey addressed to teachers who 
are currently implementing CLIL in their schools. 
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Developing a CLILTextbook Evaluation Checklist 


El desarrollo de una lista de verificacion de 
evaluacion para libros de texto AICLE 


Resumen 

En las ultimas decadas, el numero de centros bilingiies en Europa ha aumenta- 
do signifrcativamente. La mayoria de ellos implementan la metodologia AICLE, 
de forma que se crea una demanda de materiales nuevos y mas adecuados a las 
materias no linguisticas que se ensenan a traves de una lengua extranjera: se ne- 
cesitan nuevos libros de texto escritos en lengua extranjera, y la oferta existen- 
te no cubre la demanda actual. Dado que el desarrollo de materiales para AICLE 
es un campo reciente, no disponemos todavia de la investigacion necesaria enca- 
minada a la evaluacion de los aun insufrcientes libros de texto. En este articulo, 
y con el proposito de evaluar libros de texto disenados para AICLE, proponemos 
una lista de verificacion de evaluacion. Por un lado, tomamos como base listas de 
verificacion disenadas para evaluar libros de texto para la ensenanza del ingles, 
-segun Steirt and Masser, los materiales para AICLE utilizan principios/tecnicas 
procedentes de la ensenanza del ingles como lengua extranjera-. Por otro lado, nos 
basamos en los criterios de evaluacion de la produccion de materiales de calidad 
para el aprendizaje a traves de AICLE establecidos por Mehisto (2012). Esta inves¬ 
tigacion sigue en proceso con el objetivo de mejorar esta lista de prueba de cali¬ 
dad tenidendo en cuenta los resultados del analisis de los criterios de evaluacion 
llevado a cabo por un grupo de discusion formado por profesores expertos en la 
metodologia AICLE, asi como una encuesta dirigida a profesores que actualmen- 
te implementan AICLE en sus centros. 

Palabras clave: CLIL; materiales para CLIL; evaluacion; libros de texto para CLIL; 
listas de prueba de calidad para libros de textos de CLIL. 
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0 desenvolvimento de um roteiro de avaliacao 

) 

para livros didaticos da Aprendizagem Integrada 
de Conteudos e de Lingua 


Resumo 

O numero de colegios bilingues na Europa tem aumentado significativamente nas 
ultimas decadas. A maioria deles implantam a metodologia da Aprendizagem In¬ 
tegrada de Conteudos e de Lingua (AICL), de tal forma que tambem aumenta a 
demanda de novos e mais adequados materials para cada materia ensinada por 
meio de uma segunda lingua: necessitam-se textos novos escritos em lingua es- 
trangeira e a oferta existente nao atende a demanda atual. Tendo em vista que a 
producao de materials para a AICL e um campo recente, nao ha ainda sufrciente 
pesquisa dirigida a avaliacao dos ainda insufrcientes livros didaticos. Neste docu- 
mento, e com o objetivo de avaliar textos desenhados para a AICL, expoe-se ten- 
tativamente um roteiro de avaliacao de qualidade que se fundamenta em dois 
estudos. Por um lado, baseia-se em roteiros de avaliacao anteriores, desenhados 
para avaliar livros didaticos para o ensino de ingles; por outro, em criterios de ava¬ 
liacao da producao de materials de qualidade para a AICL, elaborados por Mehis- 
to (2012). Novos estudos estao em processo para melhorar esse roteiro com base 
nos resultados da analise de criterios de avaliacao que faz um grupo de abordagem 
formado por professores especializados na metodologia da AICL e mediante uma 
enquete dirigida a professores que atualmente implantam a AICL em seus colegios. 

Palavras-chave: AICL; materials didaticos para a AICL; avaliacao; livros didati¬ 
cos para a AICL; roteiro de avaliacao de qualidade para livros didaticos da AICL. 
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INTRODUCTION 

Two decades after its birth, CLIL is currently an established teaching ap¬ 
proach across Europe. Most bilingual and plurilingual schools in the Euro¬ 
pean context have adopted CLIL, and its popularity keeps growing. Teachers 
and researchers agree on the advantages this approach provides for the 
dual focus it is intended: learning language and content at the same time. 
The first two decades of implementation of CLIL show interesting bene¬ 
fits from different perspectives: students show more interest and motiva¬ 
tion than in conventional subject content classes (Dalton-Puffer & Smit, 
2007); language is learnt in more natural contexts than those of L2 learn¬ 
ing, leading to higher levels in communicative competence in L2 (Lasaga- 
baster, 2008); and thinking skills are fostered, since students make more 
cognitive effort in order to understand the content provided through L2 
(Van de Craen, Mondt, Allain, & Gao, 2007). 

The spread of CLIL approach keeps increasing and, as a consequence, 
specific pedagogic material is needed at all educational levels. On this 
respect, researchers, teachers and stakeholders frequently express the 
shortage of CLIL materials (Moore and Lorenzo, 2007; Alonso, Grisalena & 
Campo, 2008; Coyle, Hood & Marsh, 2010, among others). This situation 
makes teachers create materials adapted to their contexts, or adjust the 
few existing to their needs, which leads to an increase of their workload. 
Since the textbook is the key material that numerous teachers rely on, in 
this paper we will focus on providing a tentative checklist that facilitates 
their evaluation. 

LITERATURE REVIEW 

The use of textbooks 

Textbooks have often played an important role in the compulsory levels 
of education. Students spend much of their time exposed to them, both 
in class and working at home. Throughout decades, researchers have an¬ 
alyzed the significant role textbooks play in any generation of students’ 
training; Cast ell, Luke, & Luke (i989,vii) claim: “the school textbook holds 
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a unique and significant social function: to represent to each generation 
of students an officially sanctioned, authorised version of human knowl¬ 
edge and culture" (original emphasis). Some experts, however, have also 
questioned to what extent textbooks constitute a help or a hindrance in 
the process of teaching and learning. The debate is still present, though 
nowadays attention is turning to the use of digital or printed textbooks 
in the classrooms. 

No matter the device, teachers and learners can obtain benefits from 
the use of textbooks: they provide organization in the units or modules, 
giving an appropriate sequence of content; they cover the curriculum, and 
have been designed by experts who take into consideration the latest re¬ 
search and the teaching strategies required for each area. Students and 
parents alike usually favor to have the information organized and to be 
able to keep an easy control over the content that has already been cov¬ 
ered in class. 

In regards to well-designed language teaching textbooks, foreign 
and second language teachers -mainly in the pre-Internet era-, found key 
resources for their teaching practice in textbooks: "the extent of language 
activities worldwide could hardly be sustained without the help of the 
present generation of textbooks” (Richards, 1998, p. 127). Regarding the ad¬ 
vantages they offer, Hall (2011) summarizes: 

They provide language input and exposure for learners; they 
can provide interesting and motivating material, organized 
in an appealing and logical manner; and they provide a writ¬ 
ten record of what has been studied, allowing for revision 
and continued study beyond the classroom, (p.213) 

As for the teachers, textbooks provide them with guidance and sup¬ 
port and reduce the amount of time they require for preparation. Textbooks 
also help students to be more autonomous in their learning, since, as Ur 
(2015) points out, "a student without a coursebook is more teacher-depen¬ 
dent” (p. 198). Finally published materials have more credibility for stu¬ 
dents than teacher-generated ones (Sheldon, 1988). 

However, some counter-arguments warn about the dangers of the 
dependency on textbooks. Criticisms usually point out that textbooks can- 
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not cater for individual needs, and may not be relevant or culturally ap¬ 
propriate for a group or class. Meddings and Thornbury (2009) go beyond 
these observations and prevent against the potential material- instead of 
person-centred classes. 

The textbook, however, is a tool that becomes significant only when 
the teacher brings it to life. In order to do that, and to make the most of the 
textbook chosen, we consider the importance of being trained to iden¬ 
tify quality textbooks, since the choice can determine the quantity and 
the kind of additional resources needed for the teaching practice. Train¬ 
ing involves getting to know the principles of textbook evaluation. This 
is why we consider that textbook evaluation should constitute a signif¬ 
icant part of the teacher training in CLIL, training that is regarded as a 
priority by numerous experts (Baker, 2006; Pistorio, 2009; Hillyard, 2011, 
among others). 

CLIL and textbooks 

Current research and teaching practice show the need of specific CLIL re¬ 
sources in bilingual / plurilingual centres: "Among the most difficult issues 
regarding CLIL-style teaching is how to obtain teaching resources, includ¬ 
ing textbooks” (Zaparucha, 2009, p. 94). This situation is common across 
Europe and it is significantly relevant in Spain, where the 17 autonomous 
regions have different curriculums in their subjects, including the most 
common subjects taught through L2: social and natural sciences. The fre¬ 
quent changes in educational policies make this situation extremely com¬ 
plex for publishers and teachers alike. 

Publishers try to keep up with the bloom of bilingual centers requir¬ 
ing learning materials for more subjects every year. Focusing on the Span¬ 
ish bilingual educational context, some textbooks are specifically created 
for each region, and the implementation of the bilingual system in some 
levels or in some subjects often takes place without a previous development 
of materials needed. Since the process of textbook making takes time, and 
publishers cannot keep up to edit and plan what is needed in so many ar¬ 
eas, some teachers do not have any textbooks in L2 to rely on in the very 
first year(s) of implementation of their subject. Being the first year(s) of 
teaching through L2 the hardest ones, mainly for content teachers used to 
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teaching in their Li, the use of a textbook plays a key role as a resource in 
these situations (see Richards, 1998). British or American textbooks could 
provide some help for content subjects taught through English. Howev¬ 
er, they are not a suitable resource since the curriculum of the subjects in 
all educational levels differs from the Spanish one and also because those 
textbooks are addressed to native speakers. 

When analyzing CLIL materials, scarcity of CLIL books is the most 
common issue that stands out, though there are other aspects to be con¬ 
sidered: Meyer (2010) focuses on the lack of quality of course books; as for 
scaffolding, a key strategy in CLIL, Kelly (2010) points out the lack of lan¬ 
guage support students need in content subjects, which can be provided 
as supplementary resources of the textbooks in the form of posters, book¬ 
lets, or in the textbook alongside the task. More specifically, he observes: 
"there is a drastic need for such materials in textbooks in CLIL contexts. It 
should also go further than the provision of general academic language 
and include language functions which are specific to curriculum subjects” 
(p. 102). The relevance of cultural competence (one of the four Cs of the 
CLIL approach) is also noticed: "Studies comparing various CLIL-textbooks 
have shown that the cultural dimension has not been properly exploited 
yet” (Meyer, 2010, p. 20). On this respect, experts also observe the lack of 
connection between course books and learners’ lives (Ballman, 1997); this 
lack of connection hinders significant learning. 

Despite this situation, the role of textbooks is more important than 
ever. Unexperienced teachers are the ones who rely more on textbooks, 
and nowadays, numerous young bilingual teachers are taking up most 
CLIL vacancies, given the increasing number of plurilingual schools in the 
European context. At the same time, there is a connection between teach¬ 
ing tied to textbook and shortage of teachers (Williams, 1983, p. 251). These 
two factors: lack of experience in teaching through L2 and shortage of CLIL 
teachers, make the CLIL textbook especially significant in the current ed¬ 
ucational contexts. 

The use of checklists to evaluate textbooks 

Since CLIL materials development is a recent area of research and the pro¬ 
duction of CLIL textbooks is also evolving, evaluation of CLIL textbooks is 
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an area still to be developed. For this paper, we have created a tentative 
checklist to evaluate CLIL textbooks with the purpose of completing and 
expanding it through future research, mainly adding CLIL teachers' contri¬ 
butions. Although numerous checklists to evaluate textbooks in ELT have 
been created in the last years (see Mukundan, Nimehchisalem & Hajimo- 
hammadi, 2011, among others), research has not focused yet on the elabora¬ 
tion of a checklist to compile the main features a CLIL textbook should have. 

Checklists are considered useful instruments for teachers in order 
to select, adapt and evaluate textbooks (Sheldon, 1988; Me Grath, 2006). 
Teachers can evaluate a textbook using a checklist or impressionistically, 
i.e. just analyzing its suitability by taking a quick look at the book, which 
is a frequent procedure among experienced teachers (see Nimehchisa¬ 
lem and Mukundan, 2013). Checklists, however, are more useful when it is 
novice teachers who evaluate textbooks. Considering the number of nov¬ 
ice teachers facing the challenge of teaching content through an L2, and 
also those who are experienced teachers, but have recently started teach¬ 
ing through an L2, we consider the checklist as a very appropriate instru¬ 
ment for CLIL educational contexts. 

A textbook evaluation checklist consists of a set of criteria (items) 
that are ticked when the presence of the item is confirmed. Among its ad¬ 
vantages, Me Grath (2006) points out that it is not a watertight category; it 
contrasts system (objectivity) with impression (subjectivity), it is cost effec¬ 
tive, since a significant amount of information can be obtained in a short 
time, and it provides informants with an easy format to give information. 

Limitations of checklists have to do with the inclusion of criteria that 
might be context bound in the moment of creation, and with the need to 
adapt some of the criteria to tailor the checklist to suit a specific context. 
Demir and Ertas (2014, p. 247) summarize this fact: "no coursebook evalu¬ 
ation checklist in the literature is complete. Regardless of the number of 
items it is made up of, any checklist can be modified by adding or deleting 
items depending on the circumstances of a given instructional setting”. We 
consider that once the main basic criteria are found and fit most contexts, 
tailoring the checklist by adding some more criteria or ignoring those that 
are not suitable is an advantage of the tool itself. 
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METHODOLOGY 

In order to create our tentative CLIL textbook evaluation checklist we took 
into consideration these two areas: 

1. Review of the available instruments for ELT textbooks evaluation. 

2. Established criteria for producing CLIL materials. 

Previous research on ELT textbooks evaluation was required, first 
because CLIL materials use principles/techniques of foreign language 
teaching (Steirt and Masser, 2011), and also because CLIL benefits from the 
research done in ELT. Regarding textbook evaluation checklists specifical¬ 
ly, there is a significant research carried out in the last five decades, as re¬ 
viewed by Karamoozian and Riazi (2008), and by Mukundan and Ahour 
(2010). These authors analysed decades of validated checklists, compar¬ 
ing evaluative criteria and number of items per categories, among other 
features. Their review of instruments in order to identify well-established 
checklists eventually helped Mukundan, Hajimohammadi and Nimeh- 
chisalem (2011) to develop a tentative checklist for ELT textbooks that was 
provided to a focus group of ELT teachers in order to refine it and to give 
it more validity. We will replicate this latter design in our study. This ten¬ 
tative checklist (see appendix) will be provided to a focus group of CLIL 
teachers, former students of the Master in Bilingual Education at Univer- 
sidad Nebrija (Madrid), who will eliminate, modify and/or add criteria to 
the checklist -qualitative phase of the study. The new version of the check¬ 
list will be sent via online survey to CLIL teachers working at bilingual 
schools, who will provide feedback on the weight of each criteria. Some 
of the items might be removed after carrying out a factor analysis of their 
feedback. The information of the feedback will be included to complete 
the final version of the checklist. In order to operationalize the list, a five 
point Likert scale will be added to each criterion. Mukundan and Nimeh- 
chisalem (2012) point out that both elements (criteria and weight) are es¬ 
sential to create a complete textbook evaluation checklist. In our case, and 
for teachers' practical purposes, once the final version of the checklist is 
created, the section providing information on the weight can be ignored 
and the checklist can be used just to mark to what extent the criteria are 
present or not present in the textbook analyzed. 
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RESULTS 

The criteria have been drawn up considering the suggestions provided by 
Tomlinson, (2003) in order to avoid common mistakes in developing text 
evaluation checklists. We have first taken into consideration those com¬ 
mon features that characterize ELT and CLIL textbooks (availability, and au¬ 
thenticity of language, among others). They are included in the first section 
of the checklist. The other sections include those characteristics that re¬ 
fer mostly or totally to the content taught through L2. In order to compile 
these features, we have considered Mehisto's (2012) criteria for producing 
CLIL learning material. According to him, in the case of CLIL materials, the 
criteria required to create quality learning materials for any area should 
be taken into consideration. In addition to those common criteria, Mehis- 
to observes: "quality CLIL materials are highly integrative and multilayered 
and they help increase the likelihood that both content and language learn¬ 
ing will be meaningful” (p. 17). Among others, criteria such as making the 
learning intentions and process visible to students, fostering learning skills 
development, or including different types of formative assessment, are in¬ 
cluded in Mehisto's list. 

With regard to the structure, the checklist is made up by seven sec¬ 
tions: the first one comprises general and technical items, sections 2 to 5 
are devoted to the inclusion of the 4CS of CLIL -content, cognition, com¬ 
munication and culture-, within CLIL textbooks, section 6 includes state¬ 
ments related to language, and section 7 consists of a final statement to 
reflect upon the integration of the CLIL principles in the textbook ana¬ 
lyzed. We have tried to create a list that covers the key CLIL items with¬ 
out being too detailed or extensive, considering that it needs to be quick 
to be filled. 

The checklist can be used for primary and secondary education, since 
CLIL principles are common to all educational levels. Given the variabili¬ 
ty involved in levels, subjects and textbook approach, the checklist, as we 
have mentioned, might change in the number of criteria considered, but 
can also be altered in the weight given to each criterion. 
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CONCLUSIONS 

The tentative checklist provided in the appendix has been created consid¬ 
ering the theoretical framework of ELT textbook evaluation and on the cri¬ 
teria to develop quality CLIL materials development. In order to complete 
the process, we need to include CLIL teachers’ impressions on the list. Our 
future lines include two phases: gather information on the suitability of 
the items in order to refine the list (information provided by a focus group 
made up of CLIL teachers), and a survey, also provided to CLIL teachers via 
online, in order to analyse the weight they give to each criteria by carry¬ 
ing out factor analysis. The teachers’ contributions will provide the pro¬ 
cess with validity. Moreover, the final checklist, which does not exist on 
CLIL textbooks yet, will encompass a list of criteria to be considered when 
selecting, adapting or evaluating a CLIL textbook. Eventually, CLIL teach¬ 
ers as well as future CLIL teachers will have at their disposal a tool that 
will help them to acquire a useful and systematic insight of different CLIL 
textbooks in a short period of time. 
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APPENDIX 

Tentative checklist for CLIL textbooks 


Criteria 

I. GENERAL 

A. Structure 

1. The organization is consistent with the curriculum 

2. The objectives are specified explicitly 

3. It includes self-evaluation at the end of the modules 

4. It provides summarizing items at the end of the modules 

B. Supplementary materials 

5. The resource pack is complete 

6. The resource pack includes varied ICT resources 

7. It offers supplementary materials for under /over -achievers 

8. It provides guidance for non-native content teachers 

9. It provides support for language assistants 

C. Physical and utilitarian features 

10. It shows quality in editing and publishing 

11. Its layout is attractive 

12. It contains appropriate pictures, diagrams, tables ... 

13. It is easily available 

14. It is durable 

15. It is cost-effective 

II. CONTENT 

16. It covers the contents of the curriculum 

17. Learning outcomes for learning are specified 

18. The content is appropriate for the students’ age 

19. The content is relevant to students’ experiences 

20. The order of sequences is flexible 

21. It provides support to simplify content (scaffolding) 

22. The visual content is functional 

23. The activities suggested for practicing the content are varied 

24. The activities suggested for practicing the content are enough 

25. There is authentic material at an appropriate level. 
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III. COGNITION 

26. It allows breaking down tasks / activities to make them more manageable 
(scaffolding) 

27. It relates written work to structures and vocabulary practiced orally 

28. The activities are cognitively appropriate for the content 

29. It caters the needs of different learning styles 

30. Activities activate previous knowledge 

31. Activities are challenging 

32. Activities are motivating 

33. Activities include projects 

IV. COMMUNICATION 

34. It provides support to simplify language (scaffolding) 

35. It stresses communicative competence in activities 

36. The activities enable students to use the L2 outside the classroom situations 

37. Activities are developed to encourage teacher-student and student-student 

communication 

38. Activities are balanced between individual response, pair work and group work 

V. CULTURE 

39. It relates content to the learners' culture and environment 

40. It guides students in developing cultural awareness 

41. The content is relevant to the socio cultural environment 

42. The content involves culture-specific items 

43. The content is free from stereotypical images 

44. The visuals relate to the students own culture 

45. Cultural sensitivities have been considered 

VI. LANGUAGE 

46. The language is authentic 

47. It gives guidance in the presentation of vocabulary 

48. The number of new words in each module is appropriate to the students’level of L2 

49. There is appropriate sequencing of vocabulary (load and re-entry) 

50. It gives practice in guided composition in early stages 

51. It presents vocabulary in appropriate contexts and situations 

52. It considers proficiency level of L2 

53. It suggests aids for pronunciation 

VII. INTEGRATION 

54. It is consistent with the principles of CLIL 
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